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Using  data  from  selected  acquisition  reports,  Drezner  et  al.  (1 993)  show  that 
reform  initiatives  from  1 960  to  1 990  did  not  reduce  cost  growth  on  1 97  defense 
programs.  The  average  cost  growth  on  these  programs  was  20  percent  and 
did  not  change  significantly  for  30  years.  Using  data  from  the  Defense 
Acquisition  Executive  Summary  data  base,  we  show  a  similar  result.  Initiatives 
based  on  the  recommendations  of  the  Packard  Commission  did  not  reduce 
the  average  cost  overrun  percent  experienced  on  269  completed  defense 
acquisition  contracts  evaluated  over  an  8-year  period  (1988  through  1995). 
In  fact,  the  cost  performance  experienced  on  development  contracts  and  on 
contracts  managed  by  the  Air  Force  worsened  significantly.  Although  many 
factors  contribute  to  poor  cost  performance,  estimation  error  is  a  casual  factor 
identified  in  each  study. 


President  Ronald  Reagan  established 
the  Packard  Commission  in  1986  to 
reduce  inefficiencies  in  the  defense 
procurement  system.1  Although  the 
commission  examined  defense  manage¬ 
ment  practices  in  general,  it  focused  on 
the  acquisition  process.  The  commission 


concluded  that  the  primary  problems  with 
the  acquisition  process  were  the  same  ones 
identified  in  previous  decades  (cost  growth, 
schedule  delays,  performance  short¬ 
falls).  It  recommended  streamlining  the 
acquisition  process,  increasing  tests  and 
prototyping,  changing  the  organizational 
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culture,  improving  planning,  and  adopt¬ 
ing  the  competitive  firm  model  where 
appropriate.  Like  the  problems,  the 
recommendations  were  strikingly  similar 
to  reform  efforts  of  the  past  and  to  those 
of  the  present  (Dews,  Giles,  Barbour, 
Harris,  and  Hesse,  1979;  Gates,  1989). 

With  this  research  we  set  out  to  test  the 
effectiveness  of  the  Packard  Commission’s 
recommendations  on  reducing  defense 
cost  overruns.  As  similar  initiatives  are 
identified  and  implemented  today,  it  is 

important  for 


"Despite  the 
i  irpleirentation 
of  more  than  two 
dozen  regulatory 
and  acfrriristration 
initiatives,  there 
hasbeennostJ>- 
stantial  irrprcve- 
ment  in  the  cost 
performance  of 
defense  proyans 
for  more  than  30 
years." 


policy  makers 
to  understand 
the  effective¬ 
ness  of  past 
policies.  Based 
on  a  review  of 
269  completed 
defense  con¬ 
tracts,  we  found 
that  the  Packard 
Commission’s 
recommenda¬ 
tions  did  not  re- 


>:;i“^;;ljlllllllllilillllil!illlllilEl  duce  cost  over¬ 
runs.  This  result  is  consistent  with  simi¬ 
lar  research  involving  an  analysis  of  cost 
growth  on  197  defense  acquisition  pro¬ 
grams  (Drezner,  Jarvaise,  Hough,  & 
Norton,  1993).  Despite  the  implementa¬ 
tion  of  more  than  two  dozen  regulatory 
and  administration  initiatives,  there  has 
been  no  substantial  improvement  in  the 
cost  performance  of  defense  programs  for 
more  than  30  years. 


Literature  Review 


Unplanned  cost  increases  in  defense 
procurements  can  escalate  to  staggering 


amounts  and  can  adversely  affect  resource 
allocation  decisions,  especially  when 
defense  budgets  are  decreasing.  The 
Department  of  Defense  (DoD)  has  not 
ignored  this  problem.  As  shown  in  Table 
1,  Drezner  et  al.  (1993)  identify  several 
important  regulatory  and  administrative 
initiatives  to  improve  defense  cost 
performance.  The  expectation  was  that  as 
these  initiatives  were  implemented,  cost 
performance  would  improve  through  time. 

Unfortunately,  these  initiatives  did  not 
improve  defense  cost  performance.  Using 
data  from  the  DoD  Selected  Acquisition 
Report  (SAR),  Drezner  et  al.  (1993)  com¬ 
puted  the  average  cost  growth  on  197 
acquisition  programs  with  start  dates  from 
1960  through  1990.  Results  show  that  cost 
growth  fluctuated  around  20  percent,  with 
no  substantial  improvement  through  time. 

A  factor  contributing  to  defense  cost 
growth  is  estimation  error,  possibly  due 
to  excessive  competition.2  Contractors 
have  an  incentive  to  understate  initial  costs 
to  win  new  contracts.  Likewise,  because 
programs  compete  for  limited  resources, 
there  is  an  incentive  to  accept  low  esti¬ 
mates  as  reasonable.  Drezner  et  al.  (1993) 
could  not  test  this  assertion,  but  did  re¬ 
port  that  the  average  initial  cost  estimates 
were  systematically  understated.  This 
finding  is  consistent  with  results  reported 
by  others  (General  Accounting  Office 
[GAO],  1988;  Tyson,  Harmon,  &  Utech, 
1989;  McNichols,  McFarland,  KcKinney, 
&  Balut,  1984;  Christensen,  1994, 1996). 

There  are  two  weaknesses  with  Drezner 
et  al.  (1993).  First,  the  SAR  data  base  fo¬ 
cuses  on  defense  programs.  The  highly 
aggregated  nature  of  program  data  may 
mask  differences  that  may  otherwise  be 
apparent  at  the  contract  level.  Second,  the 
analysis  focuses  on  cost  growth,  defined 
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Table  1.  Aoqusitian  Regulations  and  Initiatives1 


Year 

Regulation  or  Initiative  Published 

1969 

Packard  Initiatives 

1971 

Blue  Ribbon  Defense  Panel  (Fitzhugh  Commission) 

1972 

DoDD  5000.1  (Major  System  Acquisitions);  Commission  on  Government 
Procurement 

1973 

DoDD  5000.4  (CAIG);  DoDD  5000.3  (T&E) 

1975 

DoDI  5000.2  (Major  System  Acquisitions);  DoDD  5000.28  (DTC) 

1976 

OMB  Circular  A-109 

1978 

Defense  Science  Board  Acquisition  Cycle  Task  Force 

1979 

Defense  Resource  Management  Study 

1981 

Carlucci  Initiatives;  Defense  Acquisition  Improvement  Program 

1982 

Nunn-McCurdy  (thresholds) 

1983 

Grace  Commission 

1985 

DoD  5000.43  (streamlining) 

1986 

Packard  Commission 

1987 

DoDD  5134.1  (USD(A);  DoDD  5000.49  (DAB) 

1989 

Defense  Management  Review 

1991 

Revised  DoDI  5000.2  (Major  System  Acquisitions) 

1994 

Federal  Acquisition  Streamlining  Act  (FASA) 

1995 

Federal  Acquisition  Improvement  Act  (FASA  II) 

1  Modified  from  Drezner  et  al.,  1993. 

as  the  difference  between  the  initial  budget  poor  control  techniques.  For  example, 
and  the  final  cost  of  the  program.  Cost  Czelusniak  and  Rodgers  (1997)  report  that 
growth  does  not  distinguish  between  Congressional  decisions  to  shift  funds  to 
uncontrollable  factors,  such  as  changes  in  near-term  priorities  external  to  a  program 
scope  or  technology,  and  controllable  (e.g.,  unplanned  contingency  operations 
factors,  such  as  inadequate  planning  or  in  Bosnia)  account  for  up  to  one-half  of 
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the  cost  growth  in  major  weapon  systems. 
Program  managers  cannot  control  this 
kind  of  cost  growth. 

Our  study  attempts  to  overcome  these 
weaknesses  by  analyzing  cost  overruns  on 
completed  defense  acquisition  contracts. 
A  cost  overrun,  defined  as  the  difference 
between  a  contract’s  final  budget  and  final 
cost,  is  a  more  appropriate  metric  for  mea¬ 
suring  the  impact  of  the  DoD  initiatives 
because  the  initiatives  focus  on  actions  that 
program  managers  can  influence,  such  as 
. I  thorough  plan¬ 
ning  and  disci- 
"The  final  budget  of  plined  control 
a  project  is  abetter  (McNaughter, 
estimate  of  wfiat  a  1990).  Funding 

well-managed  instability  and 

contact  ctoid  cost,  ch  in  re. 
because  it  includes  . 
all  the  aUUiorized  q^rements  are 
changes  that  may  typically  be- 
not  have  been  yond  the  con- 

knouin  at  the  start  trol  of  program 
of  a  contract. "  managers  and 

lead  to  cost 
.KlilllllSllIll  growth,  but  not 
necessarily  to  cost  overruns  (Christensen 
and  Gordon,  1998).  The  final  budget  of  a 
project  is  a  better  estimate  of  what  a  well- 
managed  contract  should  cost,  because  it 
includes  all  the  authorized  changes  that 
may  not  have  been  known  at  the  start  of  a 
contract. 

Another  difference  between  our  study 
and  Drezner  et  al.  (1993)  is  our  focus  on 
the  Packard  Commission’s  recommenda¬ 
tions.  Prior  initiatives  (e.g.,  McNamara, 
Carlucci)  had  been  ineffective  in  reduc¬ 
ing  cost  growth  (Dews  et  al.;  1979,  Gansler, 
1989;  Drezner  et  al.,  1993),  and  current 
initiatives  (e.g.,  the  Federal  Acquisition 
Streamlining  Act  (FASA),  the  Federal 
Acquisition  Improvement  Act)  were 


building  on  the  framework  of  the  Packard 
Commission  initiatives  (Gates,  1989). 

Methodology 


This  study  tests  whether  the  recommen¬ 
dations  implemented  as  a  result  of  the 
Packard  Commission  affected  the  cost 
performance  experienced  on  defense 
acquisition  contracts.  The  expectation  is 
that  cost  performance  would  improve  after 
implementation.  To  test  this  hypothesis, 
we  compared  the  average  cost  overrun  per¬ 
cent  on  populations  of  defense  contracts 
before  and  after  the  commission’s  recom¬ 
mendations  were  implemented  using  the 
nonparametric  Mann- Whitney  test  at  the 
10  percent  level  of  significance.3 

The  hypothesis  was  tested  using  cost 
performance  data  on  269  contracts,  com¬ 
pleted  between  January  1,  1988,  and  De¬ 
cember  31,  1995.  The  8-year  period  pro¬ 
vided  for  approximately  four  years  before 
and  after  the  treatment  date  of  December 
31,  1991.  The  decision  to  use  four  years 
of  performance  data  before  and  after  the 
treatment  date  was  subjective,  but  seemed 
adequate  to  account  for  a  gradual  imple¬ 
mentation  of  the  commission’s 
recommendations. 

Based  on  assessments  made  by  the 
GAO,  we  chose  December  31,  1991,  as 
the  treatment  date.  In  1990,  the  GAO 
reported  that  efforts  to  implement  the 
recommendations  were  in  various  stages, 
with  some  initiatives  under  way,  and 
others  planned  for  implementation  in  the 
“near  term”  (GAO,  1990).  In  1991,  the 
GAO  reported  that  most  of  the  recom¬ 
mended  changes  had  been  made  and  the 
remaining  changes  would  be  completed 
in  the  “short  term”  (GAO,  1991). 
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The  average  cost  overrun  percent 
(CO%)  is  defined  as 

CO%  =  (Final  cost  -  Final  budget)  / 
Final  budget)  ¥  100  (1) 

Average  CO%  =  ?  (CO%)  /  n  (2) 

where  the  final  cost  and  budget  of  the 
contract  were  the  actual  cost  of  work 
performed  (ACWP)  and  the  budget  at 
completion  (BAC)  reported  by  the  con¬ 
tractor  on  the  last  cost  performance  report, 


and  n  is  the  number  of  contracts.4  As  a 
relative  measure  of  cost  performance, 
CO%  adjusts  for  the  effects  of  inflation 
and  contract  size. 

The  monthly  cost  performance  report 
is  prepared  by  the  contractor  and  submit¬ 
ted  to  the  government  throughout  the  life 
of  the  contract.  Data  from  the  report  are 
stored  in  the  Defense  Acquisition  Execu¬ 
tive  Summary  (DAES)  data  base,  main¬ 
tained  by  the  Office  of  the  Secretary  of 
Defense.  At  the  time  of  this  study,  the  data 
base  contained  cost  information  from  378 


Table  2.  Surmary  of  Contract  Data 


Before  Implementation  (1 988-1 991 ) 

Phase 

Service 

All 

Development 

Production 

AF 

Navy 

Army 

Number  of  contracts  (n) 

148 

47 

101 

64 

70 

14 

Average  final  cost 
(millions  of  dollars) 

356 

312 

377 

428 

328 

174 

Standard  deviation 

1105 

598 

1276 

1581 

544 

233 

Average  final  budget 
(millions  of  dollars) 

341 

294 

363 

412 

310 

174 

Standard  deviation 

1061 

542 

1232 

1524 

505 

250 

Average  final  overrun 
(millions  of  dollars) 

15 

17 

14 

16 

17 

0 

Standard  deviation 

55 

59 

53 

60 

55 

22 

After  Implementation  (1 992-1 995) 

Phase 

Service 

All 

Development 

Production 

AF 

Navy 

Army 

Number  of  contracts  (n) 

121 

34 

87 

49 

64 

8 

Average  final  cost 
(millions  of  dollars) 

571 

768 

494 

724 

502 

190 

Standard  deviation 

1121 

2011 

426 

1674 

465 

117 

Average  final  budget 
(millions  of  dollars) 

511 

647 

458 

613 

476 

173 

Standard  deviation 

935 

1661 

390 

1379 

438 

114 

Average  final  overrun 
(millions  of  dollars) 

60 

121 

36 

111 

26 

17 

Standard  deviation 

274 

499 

84 

421 

69 

27 
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defense  acquisition  programs  encompass¬ 
ing  1,843  individual  contracts  from  1977 
through  1995.  The  reliability  of  the  data 
is  controlled  by  earned  value  management 
control  systems  criteria,5  a  DoD  require¬ 
ment  on  most  large  defense  contracts.  Vir¬ 
tually  all  of  the  data  in  the  DAES  data 
base  comes  from  criteria-compliant  con¬ 
tractors.  But  because  of  technical  prob¬ 
lems  with  the  data  base  and  our  focus  on 
completed  contracts,  we  could  only 
include  269  contracts  in  our  study.6 

We  also  evaluated  the  sensitivity  of  our 
results  to  contract  phase  (development, 
production)  and  managing  Service  (Army, 
Navy,  Air  Force).  Because  of  greater 
uncertainties,  the  cost  performance  on 
development  contracts  is  often  worse  than 
the  cost  performance  on  production 
contracts  (Christensen,  1994).  Given  the 


different  missions  and  requirements  of  the 
Army,  Navy,  and  Air  Force,  it  is  possible 
that  the  Packard  Commission’s  recom¬ 
mendations  were  implemented  differently 
across  the  Services.  Table  2  summarizes 
the  relevant  cost  data  on  the  contracts  used 
in  our  study.7 

Results 


As  illustrated  in  Figure  1,  the  recom¬ 
mendations  of  the  Packard  Commission 
affected  cost  performance,  but  in  the 
wrong  direction.  Cost  performance  wors¬ 
ened  rather  than  improved  after  implemen¬ 
tation.  As  Table  3  shows,  the  average  final 
overrun  percentage  for  contracts  before 
implementation  was  5.6  percent.  After 
implementation,  the  average  final  overrun 


Figure  1. 

Cost  Performance  Before  and  After  Inpletrenting  Packard  Reforms 
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percentage  was  9.5  percent.  With  a p-value 
of  0.055,  this  change  is  statistically  signifi¬ 
cant  at  an  a  of  0.10.8 

This  result  was  sensitive  to  contract 
phase.  The  cost  performance  on  the  sample 
of  development  contacts  worsened  from 
4.1  to  15.3  percent  (p  =  0.014).  The  cost 
performance  on  production  contracts  also 
worsened  from  6.2  to  7.2  percent,  but  the 
change  was  not  significant  (p  =  0.294). 

The  result  was  also  sensitive  to  the  mili¬ 
tary  service  that  managed  the  contract.  The 
average  cost  overrun  on  Air  Force  con¬ 
tracts  worsened  significantly  from  2.8 
percent  to  12.7  percent  {p  =  0.003).  The 
average  cost  overruns  on  Navy  and  Army 
contacts  did  not  change  significantly.  This 
was  true  regardless  of  contract  phase. 

Discussion 


Overall,  these  results  show  that  the 
Packard  Commission’s  recommendations 
did  not  improve  the  cost  performance  of 
defense  acquisition  contracts.  As  such,  the 
results  are  consistent  with  results  reported 
by  Drezner  et  al.  (1993).  Our  study  also 


shows,  however,  that  for  development  and 
Air  Force  contracts,  the  final  cost  overrun 
percentages  more  than  tripled  after  the 
recommendations  were  implemented.  In 
general,  development  contracts  are  more 
risky  than  production  contracts,  and 
appear  to  be  more  sensitive  to  policies 
affecting  cost  performance.  We  do  not 
know  why  the  cost  performance  of  Air 
Force  contracts  worsened  significantly, 
while  the  cost  performance  of  the  other 
Services  did  not  change  significantly. 
Although  the  Packard  Commission’s 
recommendations  were  to  be  implemented 
DoD-wide,  it  is  possible  that  each  Service 
implemented  them  differently.  Clearly, 
further  research  on  this  issue  is  needed. 

Drezner  et  al.  (1993)  also  reported 
strong  evidence  of  understated  initial 
budgets.  Based  on  the  same  data  base  used 
in  our  study,  Christensen  (1994,  1996) 
reported  that  the  predicted  final  cost  on 
defense  acquisition  contracts  is  also 
systematically  understated.  Thus,  not  only 
are  the  initial  budgets  understated,  but  the 
revised  estimates  of  final  cost  are  also 
understated  throughout  the  lives  of  most 
defense  acquisition  contracts.  Although 


Table  3. 

The  Effect  of  Packard  Oorrrrission  Recxxrrrendations 
on  Defense  Cost  Performance 


All 

Contracts 

Contract  Phase 

Managing  Service 

Development 

Contracts 

Production 

Contracts 

Air 

Force 

Navy 

Army 

Number  of  contracts  (n) 

269 

8 

188 

113 

134 

22 

Final  overrun  before 
implementation  (%) 

5.6 

4.1 

6.2 

2.8 

7.6 

8.1 

Final  overrun  after 
implementation  (%) 

9.5 

15.3 

7.2 

12.7 

6.1 

17.0 

Difference  (%) 

3.9 

11.2 

1.0 

9.9 

-1.5 

8.9 

Statistical  significance  ( p) 

0.055 

0.014 

0.294 

0.003 

0.206 

0.110 
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there  are  many  other  causal  factors  for 
poor  cost  performance,  it  is  apparent  that 
prior  policies  that  encourage  cost  realism 
have  been  ineffective. 

This  study  raises  some  concerns  regard¬ 
ing  the  appropriateness  of  current  reform 
initiatives.  Presently,  the  DoD  is  operat¬ 
ing  in  an  era  of  acquisition  reform.  The 
passing  of  the  FASA  in  1994  and  the  Fed¬ 
eral  Acquisition  Improvement  Act  in  1995 
marked  the  first  major  revisions  of  acqui¬ 
sition  policy  since  the  Packard  Com¬ 
mission’s  recommendations.  A  review  of 
these  policies  indicates  striking  similari¬ 
ties  between  their  major  provisions  and 
those  of  earlier  reform  efforts,  including 
the  Packard  Commission  (Gates,  1989). 
Themes  such  as  streamlining,  decentrali¬ 
zation  of  authority,  empowerment,  and 
cultural  change  simply  re-emerge  in  a 
newly  packaged  policy.  Because  the  cur¬ 
rent  provisions  are  so  similar  to  prior 
reform  efforts  that  were  ineffective,  the 
DoD  should  not  realistically  expect 
improvements  in  cost  performance. 


Conclusion 


The  recommendations  of  the  Packard 
Commission  have  been  ineffective  in 
reducing  cost  overruns  on  major  defense 
acquisition  contracts.  Cost  performance  on 
development  and  Air  Force  contracts 
actually  worse  after  implementation  of  the 
commission’s  recommendations.  Estima¬ 
tion  error  was  identified  as  one  contribut¬ 
ing  factor,  but  additional  causal  factors  are 
possible.  We  recommend  that  the  impact 
of  more  recent  policy  changes  be  investi¬ 
gated.  The  FASA  of  1994  and  the  Federal 
Acquisition  Improvement  Act  of  1995  are 
the  first  major  rewrites  of  government  pro¬ 
curement  regulations  in  a  decade.  Of 
course,  it  will  take  several  years  before 
the  cost  impact  of  these  initiatives  is 
known.  In  the  meantime,  a  comparison  of 
new  policies  with  prior  reform  efforts 
should  yield  insight  into  their  likely 
effectiveness. 
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Endnotes 


1.  The  commission’s  formal  name  was 
the  “President’s  Blue  Ribbon  Com¬ 
mission  on  Defense  Management.” 
David  Packard  was  its  chairman. 

2.  Of  course,  estimation  error  is  not  the 
only  factor  contributing  to  unplanned 
cost.  Other  factors  often  cited  in  the 
literature  include  schedule  slips, 
changes  in  requirements,  improve¬ 
ments,  management  inefficiencies, 
and  organizational  culture  (Scott, 
1983). 

3.  The  parametric  t-test  failed  the  key 
assumptions  of  normality  and  equal 
variances.  For  a  description  of  the 
Mann-Whitney  test,  see  Conover 
(1980). 

4.  To  simply  the  technical  jargon,  we 
prefer  the  term  “cost  overrun”  to 
“adverse  cost  variance.”  An  adverse 
cost  variance  occurs  when  ACWP 
exceeds  the  flexible  budget,  termed 
the  “budgeted  cost  of  work 
performed”  (BCWP). 

5 .  EM V S  criteri  a  were  formerly  termed 
“cost/schedule  control  systems  crite¬ 
ria”  (C/SCSC).  In  1996,  the  criteria 
were  slightly  revised  and  renamed. 
Despite  the  multiple  names,  the  crite¬ 
ria  have  not  changed  significantly 
since  their  inception  in  1967. 


6.  A  contract  was  included  in  our  analy¬ 
sis  if  it  was  completed  within  the  8- 
year  period,  and  the  necessary  data 
were  available  (contract  phase, 
ACWP,  BCWP,  and  BAC).  The  DAES 
data  base  that  we  used  did  not  include 
classified  programs.  See  Searle  (1997) 
for  a  complete  listing  of  the  data  used 
in  this  study. 

7.  This  study  has  limitations  that 
threaten  its  internal  validity.  In  par¬ 
ticular,  political,  macro-economic, 
and  technological  events  may  have 
occurred  during  the  8  year-period  that 
may  have  influenced  contract  perfor¬ 
mance.  We  made  no  attempt  to  con¬ 
trol  for  such  events.  In  addition,  our 
sample  was  not  evenly  distributed 
across  the  Services,  with  only  8  per¬ 
cent  of  the  contracts  managed  by  the 
Army.  It  is  possible  that  the  uneven 
distribution  may  have  biased  our 
results. 

8 .  All  hypothesis  tests  in  this  study  were 
directional  or  one-sided. 
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